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ABSTRACT;  '  .. 

* 

ThJa  resort  descHtei  work  dona  In  Imm#  processing  using  an 
Interactive  compute*  system,  Ttcnnloues  for  image  differencing  Arc 
ditch  load  mi  exanoin  using  jmaijei  returned  f  rOn  Hart  by  the  Marine, 
uln«  spacecraft  tr«  Also  aesorlotd  mr*  technique!  for  stereo 

liragi  Prpoeselng,  Stereo  areo*ulng  for  both  conventional  oamera 
systems  ana  the  Vising  1975  Leaser  camera  ayitem  la  reviewed. 
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A,  INTRODUCTION 


This  report  reviews  the  worK  of  the  Stanford  Artificial 
Intelligence  Laboratory  done  under  NASA  Grant  NGR-05-020-508 ,  For 
the  purpose  of  this  report  the  work  Is  divided  Into  three  areas: 
Image  Information  management,  automated  Image  differencing,  and 
stereo  ImaJ'e  processing,  Section  B  discusses  some  of  the  problems 
Involved  with  handling  a  larse  volume  of  Image  data  and  some  of  the 
solutions.  Section  c  reviews  the  Image  differencing  work  together 
with  various  Input  Processing  steps  used  In  preparing  the  data  for 
differencing,  section  0  describes  work  done  In  the  area  of  near-fleld 
stereo  Image  analysis  oriented  towards  the  Viking  1975  lander  camera 
system,  Appendlcles  A  and  0  are  two  term  papers  related  to  the 
ouestlor  of  stereo  image  processing  whloh  were  supported,  In  part,  by 
this  grant,  The  data  bast  used  In  this  work  has  oome  from  ths  Mariner 
Mars  probes  of  1969  and  1971,  from  prototype  Viking  1975  camera 
eauloment,  and  from  locally  produced  (mages  of  earth  soenes. 

In  addition  to  the  above  mentioned  grant,  this  work  redeyes 

support  from  jpl  Contract  952489,  Langley  Contraot  NAS  1-9682,  and 
ARPA  Contract  SD-I83. 


B,  image  information  management 


An  Information  retrieval  capability  Mas  bean  Implemanttd  at 
the  Stanford  A/I  Project  wh|oh  enables  us  to  Quiokly  revjaw  the 
p|anet  ooveraga  of  *he  MM-71  TV  Mission,  I*  Is  primarily  oriented 
toward  repealing  the  extent  of  repeated  TV  oovsrags  pf  any  area 
specified  by  latitude  and  longitude,  It  enables  the  user  to  oulokly 
determine  if  an  area  has  bsen  photographed,  and  If  so,  how  many 
times.,  on  which  orbits,  by  whloh  camera,  and  by  which  clotures  within 
an  orbit,  On  a  display  screen  Is  shown  the  disk  of  the  planet,  the 
footprints  of  the  Images,  and  vpotors  Indicating  view  and  sun  angles, 
The  correspondence  between  OAS  shutter  time  and  the 
orb  I t-oamera-p I cture  within  orbit  (Experimenter)  Identifier  le  also 
shown,  The  user  is  also  able  to  alter  the  soale  of  the  display  to 
Improve  clarity,  Most  of  the  Input  data  for  this  system  oomee  form 
the  MM' 71  LIBSET  system  operated  by  the  Solehoe  Data  Team  (SDT)  at 
JPL  • 


As  additional  tql  (Preliminary  navigation  parameters)  Pioture 
catalog  data  Is  received  It  merged  with  the  data  previously  received 
and  stored  on  the  disk  system,  This  operation  provides  us  with  the 
navigation  data  nsoessary  to  perform  the  geometric  projections 
described  below  end  also  provides  the  library  Information  used  to 
locate  Images  on  the  Image  data  (PTV)  tapes  and  the  JPL  film 
products,  The  ex*ct  manner  In  whloh  ths  above  has  been  dons  has 
varied  during  the  mission  elnee  the  reliability  and  tlmsllnses  with 
which  w§  have  received  the  TQL  and  Picture  Catalog  data  tapes  has 
varied,  we  began  receiving  the  Picture  Catalog  data  on  tape  near 
the  end  of  the  nominal  mission  primarily  In  an  attempt  to  make  up  fai¬ 
ths  ■6i*nes  of  m*  complete  SEflR,  SDT  also  incfudis  the  IPl 
Ennamcerant  loo  on  the  tape  -high  Allows  us  to  automatically  review 
the  RQH  (final  dags \ I  brat  I  on  processing!  status, 

as  additional  data  on  the  Images  la  more  readily  available, 
It  win  0*00*1  a  pa,rt  of  the  system.  In  the  short  time  that  the 
system  hat  been  operational  It  has  proven  to  be  a  valuable  asset, 
since  more  teen  s|x  thousand  image*  ex  I  at,  the  need  for  iuon  a  system 
Is  oo v | pu i ,  Its  f mporfinfl*  will  ore w  at  the  number  of  Images, 
and  the  Information  about  thej*  Tmagae,  Fnor*ea*s,  This  caoablllty 
ooulo  prove  cult*  useful  In  pioture  targeting  and  lending  alts 
Solectron  for  the  Vlhlng  mliefon. 

It  It  Important  to  note  that  this  le  an  Interactive  system 
oriented  towards  t*t  needs  of  the  sc  lent  lets,  Its  success  depends  on 
Its  ability  to  present  date  In  a  manner  oonelstant  In  format  and 
organization  with  the  way  the  experimenters  v|aw  tha  objeot  under 
Investigation, 

The  above  mentioned  oapablllty  actually  represents  the 
Initial  phase  of  the  ppooess  for  the  orojeotlon  and  dlfferenolng 
operation,  with  the  Identifiers  and  footprints  of  ail  ths  Images 


before  the  user  the  list  can  be  pruned  until  Just  the  footprints  of 
interest  are  present,  The  user  can  then  proceed  dlreotly  to  the 
projection  and  differencing  steps. 


One  area  which  deserves  additional  attention  is  the 
cataloging  of  output  products,  when  simple  operations  are  performed 
on  a  single  Image  it  Is  generally  aulte  easy  to  oata|ofl  the  output 
products,  The  problem  becomes  somewhat  complex  when  several  images 
are  combined  as  In  the  case  of  Image  differencing*  oo  or 
reconstruction,  and  polarization  studies,  The  approach  we  are  using 
Involves  the  use  of  a  header  of  arbitrary  size  whloh  is  stored  with 
every  processed  Image,  Actually,  two  headers  are  used)  one  Is  a 
"short  form"  which  simply  indicates  the  Images  and  processes  used  to 
create  the  Image  |n  Question,  the  other  Is  a  "long  form"  which 
Includes  the  actual  parameters  used  In  eaeh  of  the  Processes,  The 
beniflts  of  such  a  system  come  only  with  the  ability  to  oulckly 
retrieve  Images  utilizing  the  Information  In  these  headers  and 
Information  about  the  original  Imagas  (looatlon  on  the  planet, 
camepa,  filter,  shutter  tl^e  and  date,  prblt,  and  eto), 


The  above  capacity,  when  combined  with  a  dlso  based  storage 

system  (see  Incut  processing  be|ow),  gives  the  scientist  a 
significant  degree  of  flexibility  to  review  the  Image  data  and  the 
processing  carried  out  on  It, 
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C,  AUTOMATED  IMAGE  DIFFERENCING 


Input  Processing 

The  processing  o'  a  aarles  of  Images  Is  generally  Initiated 
by  an  Invsstlgator  supplying  us  with  a  list  of  ototurs  Identifiers 
(either  DAS  time  or  orbit  and  oloturs  within  orbit)  of  ths  Imagss  In 
which  he  is  Interested. 

The  Identifiers  are  entered  at  a  terminal  and  a  search  of 
library  Information  Is  made  to  determine!  |f  navigation  data  Is 
available  for  the  Images,  If  the  PTV  or  RDR  tapes  for  the  Images  are 
available  In  our  library.  If  the  Images  have  already  been  loaded  Into 
our  disk  system  (see  below)  as  the  result  of  prsvious  Prooessing,  and 
other  Information  relatsd  to  the  Image  (filter,  exposure  time,  and 
stc) . 


If  the  Image  has  not  been  previously  oroosseed  but  Is 
available  in  our  tape  library  It  Is  mounted  on  a  taps  unit  and  read 
In,  Several  operation  are  applied  to  the  data  at  this  time.  A  "first 
order "  photometric  correction  to  made  using  a  two  dlmsnslonal 
Interpolation  of  a  matrix  of  vldloon  response  parameters,  Ressau 
marks  are  also  located  and  square  areas  (whloh  approximate  their 
actual  shape  In  an  Image)  are  set  to  a  zero  DN  value,  thus  Identlflng 
those  points  as  Invalid  data  for  later  operations,  Next,  a 
"custer I ng"  operation  Is  performed  to  identify  bit  droppages  In  ths 
Imaos  data,  This  Is  done  by  oomparins  ernh  point  to  the  mean  of  a 
three  by  three  area  around  It,  A  rilftorenoo  of  more  than  three 
standard  deviations  from  ths  mean  Identifies  It  as  and  error,  This 
procedure  also  has  the  effect  of  Identifying  as  errors  the  various 
"fringes"  left  as  the  result  of  modeling  the  Images  of  the  reseaue  as 
squares,  Finally,  these  pixels  Identified  as  errors  art  reoiaosd  by 
averaging  the  neighboring  points,  Ths  r**ult  Is  an  Image  with  a 
reasonable  degree  of  ohotometrlo  Integrity  (slOnjfloant  errors  still 
exists  along  the  edges)  with  rsseaus  shown  as  square  arrays  of 
I nva lid  (zero)  data, 

As  these  processes  are  being  applied,  the  Image  Is  bslng 
transferee  from  the  tape  to  the  gensral  system  disk  area,  One 
additional  operation  takes  Place,  The  successive  dlffersnoes  bstwesn 
each  pixel  and  Its  neighbor  on  the  left  Is  cajoulated  and  a  histogram 
of  all  th«ss  values  is  developed.  This  histogram  Is  used  to  develop 
a  modified  Huffman  coding  sohama  for  tha  dlffersnoes.  This  Is  a 
variable  length  coding  system  with  whloh  we  are  able  to  compress  the 
Images  by  a  faotor  of  bstwesn  two  and  three  (muoh  better  on  satellite 
Images)  without  any  Information  lose,  As  this  oomoreasien  is  being 
done  ths  resulting  data  Is  stored  under  our  User  Disk  Paok  (UDP) 
arrangement  on  tha  IBM  3330  disk  system,  This  facility  allowe  a 
user  to  have  an  exchangeable  disk  paok  for  hie  own  uss,  He  expect  to 
be  able  to  store  about  200  oomplets  (mages  on  a  single  disk  paok 


using  the  aoove  compression  scheme, 

The  user  Is  thus  left  with  the  compressed  version  on  the  UDP 
for  future  reference  (after  decompress  I  on)  end  the  normally  ooded 
version  on  the  genera|  file  system,  The  normal  one  Is  used  as 
described  below  to  satisfy  the  current  prooesslna  request  and 
afterwards  Is  deleted, 


I  mage  D 1 f f e r enc I ng  < l ) 

Two  images  are  differenced  by  transforming  them  to  the  same 
projection,  aligning  them,  and  then  subtracting  the  aligned  Images, 

The  user  Is  able  to  select  any  portion  of  the  intersection  of 
two  Images  for  differencing,  Actually,  what  Is  selectd  Is  a 
rectangular  window  In  an  orthographic  Projection  for  whloh  there  Is 
data  In  ooth  Images,  Thus,  two  new  Images  ere  oreated,  However,  they 
are  not  accurately  aligned  due  to  errors  In  the  navigation  (TQL)  data 
that  Is  used  In  developing  the  projections,  These  errors  In 
alignment  are  determined  by  successively  aligning  subareas  of  the 
window  using  a  cross  correlation  technique,  For  eaoh  sub-area  an 
error  vector  Is  thus  determined,  One  of  the  original  Projections  Is 
then  repeated  Incorporating  this  error  Information  and  the  result  Is 
two  Images  that  are  generally  In  alignment  to  within  a  pixel, 

These  two  aligned  Images,  oa I  I  them  A  and  8,  are  then 
subtraotd,  The  difference  Images  A-B  and  B-A  art  oriated  and 
displayed  on  a  standard  TV  monitor  together  with  the  Images  A  and  B, 

Output  Products 

The  experimenter  Is  then  ab | e  to  make  4"  by  5”  Polaroid 
prints  of  the  images  displayed,  Also  produoed  Is  a  computer 
orirt-out  describing  the  processes  by  which  these  Images  were 
generated  and  the  parameters  used  In  eech  prooess.  This  log  Is 
maintained  automatically  by  the  Individual  prooesslng  steps  and 
resides  on  the  disk  system, 

The  capacity  also  exists  to  put  these  resulting  Imagesi  and 
notations  describing  them,  onto  magnetic  tepe  In  a  form  that  oan  be 
reao  by  the  JPL-IPL  Video  Film  Converter  for  production  of  hard  copy. 
These  products  are  then  entered  Into  the  Science  Data  Team's  data 
library. 


(1)  See  "Computer  Comparison  of  Pictures”,  Lynn  H,  Quam,  Stanford 
Artificial  Intelligence  Project  Memo  AIM-144,  May  I97I, 
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Examples  of  Image  Differencing 

Discussed  below  are  some  examples  of  Images  whloh  have  been 
projected,  aligned,  and  differenced  using  the  ttehnlaues  dlsscussed, 

FIGURE  1  shows,  at  the  top,  portions  of  two  high  resolution 
Mariner  9  Images  taken  of  the  Thyles  Mens  region  of  Mars  (75  deg,  S», 
165  deg,  w,>,  These  were  taken  under  almost  Identical  Illumination 
and  viewing  ang|es  eighteen  days  apart  (upper  left  on  Orbit  U3, 
upper  right  on  Orbit  150),  The  upper  frames  show  the  the  (mages  after 
they  have  been  transformed  to  a  common  Projection  and  aligned  as 
previously  described,  Even  with  this  processing  dons,  It  la  dlffloult 
to  accurately  determine  the  change?  that  have  taken  biace,  The 
bott°m  f Pame8  ahow  th«5r  dlfferenoe  palrs*  loft  minus  rl3ht  and  right 
minus  left,  The  subtle  ohanges  which  nave  occurred  In  the  "riffles" 
are  clearly  brought  out  In  these  lower  frames, 

Figure  2  Is  an  example  of  a  remarkable  ohange  In  a  dark  tall 
strikingly  brought  out  by  the  Ploture  differenolng  techniques.  These 
Images  of  the  northarn  part  of  Thaumasla  were  taken  nineteen  days 
apart  under  s I m | I  I ar  lighting  conditions  and  viewed  near  vertlole  but 
from  oooslte  directions,  Note  the  email  blaok  tall  In  the  {upper  right 
of  the  area,  Since  It  has  not  ohanged  It  Is  oleanly  removed  by  the 
picture  differencing  prooesa, 

Two  views  of  a  portion  of  Pyrrhae  Reglo  appear  In  Figure  3 
and  show  some  rather  obvious  ohanges,  Dark  material  has  appeared 
along  scarps,  crater  walls,  and  other  topographical  boundaries. 
Although  the  major  changes  In  the  top  frames  are  easily  detectable  by 
the  eye,  tha  minor  ones  emerge  olearly  only  In  the  dlfferenoe  Images. 

In  Figure  4  Is  another  portion  of  Pyrrhae  Reglo,  Again,  dark 
material  has  appeared  along  a  crater  wall,  These  examples  are  from 
the  same  original  Images  ae  those  In  Figure  3  und,  like  Flour*  3, 
have  slml I lar  Illumination  oondltlons  but  viewed  In  opposite 
dlreotlons  from  near  the  vertlole,  The  toppographlc  features  are 
completely  removed  In  the  picture  difference,  leaving  only  the  true 
albedo  changes, 
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D.  VIKJNG  LANDER  IMAGERY  INVESTIGATION 


The  effort  has  been  devoted  to  problems  essoolated  with  the 
extract I0n  of  near -f  laid  Panglng  |nformatl0n  from  lander  oemera 
sterao  Image  pairs,  This  latter  work#  whloh  has  used  the 
facilities  of  the  Stanford  Artificial  Intelligence  Laboratory#  Is 
described  below, 

Figure  i  Illustrates  a  stereo  pair  of  Images  rsoorded  of  a 
portable  terrain  model  by  a  lander  camera  prototype,  Ths  figures 
displayed  In  this  and  ths  following  Illustrations  are  reproductions 
of  Polaroid  Pictures  taken  of  the  output  of  a  computer  driven 
vldeo-synthesUer ,  Ths  upper  and  lower  Images  represent  left  and 
right  views,  r esoect I ve I y,  of  the  model  as  rsoorded  by  a  single 
lander  camera  prototype  looated  at  two  different  positions  100  om 
apart#  approximately  135  om  above  the  model  and  at  rouohjy  100-200  om 
horizontal  range.  The  model  oonslsts  of  an  undulating  surfaes  of 
dark  sand  upon  which  have  bean  plaoed  four  oonsplcuous  rooks  and  soms 
smaller  pebbles.  since  the  horizontal  range  Is  oomparabls  to  ths 
Inter-camera  displacement,  henceforth  called  the  ”basa|Jne”,  the  two 
views  of  the  scene  appear  appreciably  different,  It  has  been 
determined  experimentally  that  It  is  Impossible  to  visually  fuse  suoh 
disparate  Images,  a  prerequisite  to  visual  depth  peroeptlon, 

Figure  2  Indicates  the  st*me  pair  of  Images  upon  which  have 
been  overlayed  two  smooth  curves.  These  ourves  have  been  oonstruoted 
as  follows,  A  Plane,  henceforth  referred  to  as  the  "oamer a-oenter s 
plane”.  Is  defined  to  oontaln  the  two  effeotlve  oamsra-oenter 
positions.  This  plane  has  been  rotated  about  the  baseline  until  it 
passes  through  the  selected  point  of  Interest  in  the  left  view  at  the 
oenter  of  the  orepos I t I oned  box  located  directly  beneath  the  oentral 
rock,  This  plane  projsgts  Into  the  left  and  right  oarnsrc  views  as 
the  curves  shown,  The  fact  that  the  plane  projeots  as  a  eurvs 
rather  than  a  straight  line  Is  a  oonsedusnoe  of  the  eoannlng  geometry 
of  the  facsimile  camera  and  of  the  associated  display  format. 
Specifically,  ths  camera  scans  the  soene,  point  by  point,  in  uniform 
polar  and  azimuthal  steps,  The  display  format  le  linear  In  polar  pnd 
azimuthal  angles,  and  henoe  represents  a  linear  mapping  of  the 
original  reoordlng,  The  lowest  point  In  eaoh  of  these  ourves 
corresponds  to  an  azimuthal  viewing  dlr«ot|on  perpendicular  to  the 
baseline  at  *he  camera  location  In  qyest)on,  Soene  points  lying  on 
the  camera-centers  plane  anu  oommon  to  both  views  must  nsoessarily 
lie  on  the  projections  of  this  plane  In  eaoh  Image, 

Figure  3  Illustrates  the  same  Image  pair  overlayed  with  Image 
point  location  boxe8,used  In  the  soene  ranging  mode,  The  box  In  saoh 
view  has  been  visually  positioned  to  a  soene  point  that  is  common  to 
both  views  and  Is  to  be  ranged,  The  positioning  Prooedura  is  as 
follows,  The  box  |$  first  Interactively  oentered  about  a  point  of 
interest  In  the  left  oamara  Image,  A  oamsra-oenters  plans  Is  then 
tilted  to  contain  this  pointing  dlreotlon,  The  physical  point  of 
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interest  In  the  right  view  now  must  lie  on  the  projection  of  the 
Plane  In  that  Image,  Thus»  only  a  single  degree  of  freedom  remains 
for  the  definition  of  the  corresponding  box  location  In  the  rlflhthand 
view,  This  latter  parameter  has  been  arbitrarily  chosen  to  be  the 
x-coordlnate  of  the  point,  In  Image  coordinates.  The  looatlon  of  the 
matching  point  In  the  right  view  Is  determined  visually,  The  box  Is 
then  brought  to  position  by  the  adjustment  of  the  x«ooor d ! nate ,  The 
associated  value  of  the  y-coor d I nate  of  the  point  la  then  Immediately 
evaluated,  remaining  parameter. 

Once  the  corresponding  pointing  directions  In  the  two  soenee 
have  been  established,  we  have  sufficient  Information  to  evaluate  the 
spatial  location  of  the  selected  point  relative  to  the  oameras,  The 
ranging  Information  Is  promptly  computed  and  reoorded  and/or 
d I  splayed. 

Following  development  of  the  above  Programs  we  moved  on  to  a 
familiarization  study  of  some  of  the  char aoter I st I cs  of  ths  unloue 
data  format,  preparatory  to  Investigating  automation  of  ranging  and 
contour  nap  production, 

A  first  step  In  this  process  Is  Indloated  In  Figure  4,  which 
contains  additional  overlays  to  those  discussed  above,  The 
oscillatory  curves  appearing  In  these  v|ewe  represent  plots  of  the 
scene  intensity  scanned  along  the  camera-centers  plane  end 
parameterized  linearly  In  the  x-coor d I nate ,  It  will  be  noted  that 
the  lines  of  constant  phase  for  the  sand  ripples  running  aorose  the 
lower  lefthand  portion  of  the  left  view  are  roughly  orthogonal  to  the 
baseline,  These  same  waves,  when  viewed  from  the  rlghthand  camera 
position  are  observed  obliquely  and  consequently  exhibit 
foreshortened  projected  wavelength,  The  relative  dletortlone 
associated  with  the  grossly  different  perspectives  of  the  the  twp 
views  pose  special  Picture  point  correlation  problems  for  automation 
of  ranging, 

Figure  5  Illustrates  a  remapping  of  the  Intensity  plots  of 
Figure  4  Into  a  format  that  would  be  more  amenable  to  a 
one-d I  mens  I ona  I  correlation  of  data  from  the  two  scenes,  The 
intensity  curves  in  Figure  5  have  been  obtained  by  transforming  the 
curves  In  Figure  4  to  the  apoearance  they  would  have  if  reoorded  by  a 
camera  located  at  the  mld-oolnt  between  the  left  and  right  oamera 
positions,  under  the  assumption  that  the  scene  Is  perfeotly  flat, 
horizontal,  and  at  the  nominal  value  of  observed  relative  elevation. 
The  horizontal  scale  has  been  changed  to  utilize  the  full  width  of 
the  screen,  The  mapping  between  the  scene  and  the  Image  now  |« 
nonlinear  and  the  mutual  Interrelationship  no  longer  Is  as  readily 
dlscemable  as  In  the  previous  Illustrations,  It  Is  evident, 
however,  that  one-dimensional  picture  point  correlation  would  be  more 
reaolly  conducted  In  this  transformed  Image  Intensity  space  than  in 
the  Initial  space, 

we  do  not  Plan  to  proceed  further  along  the  above  lines. 


Rather*  we  are  considering  a  mora  genaral  and  powerful  approaoh 
toward  tne  development  of  near-fleld  automated  ranging,  The  proposal 
Is  to  explore  the  portion  of  the  3-space  mutually  aooeseable  through 
the  left  and  right  windows,  by  correlation  of  left  and  right  Imagery 
data  over  a  variously  tlltad  and  positioned  probing  planar  patoh. 
Once  It  has  baen  determined,  by  presorlbed  correlation  orlterla,  that 
we  have  succeeded  In  "landing'*  on  a  surfaoe,  we  oan  "crawl"  over  the 
surface  in  any  manner  desired,  accumulating  ranging  Information  as  we 
go,  Elevation  contour  lines  could  for  example  be  generated  by 
Instructing  the  props  to  explore  at  fixed  elevation,  An  automated 
contour  map  could  thus  be  constructed, 

we  also  ar>  oommmenclng  various  Image  transformations 
directed  both  toward  oomoensatlng  for  Inherent  Projeotlve  dlstortlone 
and  toward  facilitating  the  comparison  both  of  (mages  taken  from  the 
same  camera  under  different  recording  oondltlons  and  of  Images  taken 
from  the  two  different  lander  oamera  positions,  FIGURE  CAPTIONS 
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Figure  5.  A  stereo  pair  of  images  recorded  of  a  portable  terrain 
model  by  a  single  lander  camera  prototype  located  at 
two  different  positions.  Upper  image  left  view; 
lower  image  right  view. 
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Figure  6.  Overlay  of  a  projection  of  the  camera-centers  plane  onto 


the  stereo  images. 
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Figure  7.  Stereo  images  overlayed  with  image  point  location  boxes 
in  the  scene  point  ranging  mode. 
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Figure  8.  Overlays  of  the  scene  point  intensity  along  the  camera- 


centers  plane  as  a  function  of  the  x-coordinate  of  the 
image  point. 
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Figure  9.  Overlays  of  transformed  scene  point  intensities. 
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preface 


This  pep nr  reports  progress  made  on  a  system  of  programs  for 
processing  medium-angle  3tereo  Images,  The  paper  takes  the  form  of 
documentation  on  wnat  the  seoarate  programs  written  for  this  project 
do#  with  comments  as  to  how  they  fit  together,  None  of  the 
descriptions  aro  Intended  to  enable  the  casual  reader  of  this  oaoer 
to  use  the  programs  Involved,  Anyone  desiring  to  operate  any  of  these 
programs  Is  ndvlsed  to  contact  the  author  for  e  demonstration, 


I Ntr0ouCti On 


Suppose  one  were  given  two  pictures  of  the  seme  scene  taken 
from  moderately  different  viewing  points*  By  moderately  different  is 
meant  that  the  change  In  view  point  oauses  the  pictures  to  differ  by 
more  then  an  Infinitesimal  amount  but  not  by  so  much  that  an  object 
oresent  in  both  pictures  is  not. easily  recognizable  as  being  the  Same 
object,  Mathematically#  this  can  be  characterized  by  thinking  of  the 
focal  axes  of  the  two  cameras  as  vectors  and  describing  «,  the  angle 
between  these  vectors,  for  the  purposes  of  this  report,  |a|<«r/8  Is  r 
reasonable  approximation  to  the  ohras#  "moderately  different  viewing 
do  I rts" , 


G|ven  two  such  pictures,  one  would  like  to  know  how  they 
relate  to  one  another,  How  were  the  cameras  that  took  them  arranged 
with  resnect  to  each  other?  What  clues  are  there  In  the  two  pictures 
as  to  size  and  position  of  the  objeots? 

Several  things  are  known  to  be  undeoldable  given  Just  the 
Information  In  the  pictures.  Absolute  position,  for  Instance*  le  not 
derivable,  that  Is#  It  Is  not  possible  to  say  precisely  where  In 
3-space  one  of  the  cameras  was  or  to  give  the  exact  thres-d'l  mens  I  one  I 
co-ordinates  corresponding  to  a  given  point  In  a  picture,  Likewise, 
It  Is  Impossible  to  say  exactly  how  large  or  how  far  away  a  given 
obJeet  is,  Both  absolute  position  and  absoluta  size  reaulre 
knowledge  not  contained  In  the  pictures. 

It  Is  possible*  however*  to  derive  relative  positions  and 
relative  sizes  for  objects  In  the  pictures,  This  la  done  by 
assigning  an  arbitrary  position  and  orientation  to  one  of  the  oameras 
and  by  fixing  some  distance#  usually  the  baseline  distance  between 
cameras.  From  these  starting  points,  the  orientations  of  the 
cameras  and  positions  of  objects  which  appear  In  both  ploturss  can  be 
calculated. 

This  project,  then#  was  a  start  toward  automating  the 
calculation  of  said  relative  orientations  and  positions  given  no  more 
than  two  stereo  views  and  a  reasonable  guess  as  to  the  baseline 
d i stance , 


THE  PROGRAMS 

Work  on  this  project  was  segmented  Into  separate  tasks#  each 
performed  by  an  Independent  SAIL  program,  This  segmsntatlon  was 
force*  to  some  extent,  by  the  fact  tnat  the  system  usually  doss  not 
live  long  enough  to  support  one  long  program,  Thus  It  beosme 
advisable  to  have  several  small  programs  whloh  depended  on  user 
Interaction  rather  than  one  large  program  which  would  run  by  Itself, 

The  basic  sections  and  their  funcclons  are  PARSET#  whloh 
flnas  pairs  of  Points  using  no  outside  information  about  the 


la 


Djctures,  CAMERA,  wh|Ch  usas  po|nt-p6|rs  to  f|nd  approximate  camera 
m0dels*  and  CAMSCH,  which  finds  0alPs  0f  D0lnts  using  camera  m0dela* 

PARSET 


The  purpose  of  this  program  Is  to  find  a  set  of  pairs  of 
points,  one  point  out  of  9ach  picture,  which  match,  Intuitively*  two 
points  natch  If  thev  noth  are  Projections  of  the  same  three- 
aimerslonal  point,  Computationally,  the  criterion  for  mateh  la  that 
the  norna , I  zed  cr 0ss-cor r e ,at i 0n  betWee^  the  2n+l  x  2n+l  wlnd0ws 
immeolately  surroundlno  each  of  tne  two  ooln^s  be  high  enough,  Since 
the  computational  process  can  only  say  that  point  A  matches  oolnt  B 
with  probability  P,  this  program's  purpose  is  to  find  pairs  of  points 
which  match  with  fajrly  high  probability, 

As  a  preliminary  to  the  matching  process*  this  orogram 
segments  both  pictures  Into  overlapping  areas*  usually  20  pixels 
sauare.  it  then  computes  the  mean  and  variance  of  each  area  In  each 
picture  and  sorts  each  picture's  areas  by  variance*  Keeping  track  of 
where  In  the  picture  eacn  area  came  from, 

The  matching  process  begins  by  selecting  an  area  at  random 
from  the  top  end  cf  the  variance  list  of  the  first  picture,  usually 
the  top  25%,  This  limitation  Is  imposed  because  the  measure  of  match 
oelnr  used--  normalized  -ross-eor re  I  at  I  on--  works  best  where  there  is 
a  large  amount  of  information  present,  which  is  symotomlzed  by  the 
variance  being  larPe. 

Since  areas  which  match  should  have  similar  variances*  the 
selected  area  of  the  first  picture  Is  compared  wlth  each  area  of  the 
second  picture  whose  variance  Is  within  20%  of  that  of  the  area  under 
consideration,  (In  the  following,  let  the  prefixes  ’’first-”  and 
"second-”  stand  for  tne  modifying  phrases  "of  the  first  olcture”  and 
"of  the  second  picture"  respectively,) 

Each  eliylple  second-aree  Is  Initially  tested  to  see  If  (ts 
mean  is  similar  that  of  the  first-area.  If  a  second-area  basses  this 
test*  a  search  is  made  to  find  the  second-point  (some  oolnt  In  or 
near  the  second-area  unde.r  consideration)  such  that  the  2n+l  x  2n+l 
wlnccw  surrounding  this  second-point  Is  the  best  match  for  (hen  the 
highest  normalized  cross-  correlation  with)  the  2n+l  x  2n+l  window 
surrounalng  the  center  point  of  the  first-area.  The  search  strategy 
usea  Is  essentially  that  used  by  Quam  (1971), 

For  computational  expediency,  the  above  search  Is  carried  out 
usir.o  a  small  wlrdow,  typically  7  pixels  sauare,  As  the  program 
proceeds  through  the  second-areas,  the  N  second-areas  (where  N  Is 
usually  5)  yieldlnp  the  nighest  correlation  values  are  keot  track  of 
and  are  re-s®arched  using  a  larger  area*  typically  21  pixels  sauare, 
to  check  that  the  area?  do  indeed  match, 
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Tests  are  then  made  to  determine  whether  the  best  match  found 
was  nood  enough,  First  of  all#  the  correlation  must  be  abJVe  ,5, 
since  a  correlation  lower  than  ,5  can  occur  between  areas  whloh  do 
not  really  match,  Secondly#  the  too  two  correlations  must  differ 
significantly.  Failure  to  do  so  would  Indicate  that  more  then  one 
match  was  possible*  casting  doiibt  on  the  validity  of  either  match. 
Failure  In  either  of  these  tests  causes  a  first-area  to  be  rejected 
as  having  no  reliable  match*  and  another  first-area  Is  tried, 

Note  that,  when  this  process  Is  finished*  the  center  point  of 
the  first-area  has  been  paired  with  a  Sfccond-polnt  which  has  Integer 
co-ordinates.  In  practice*  however,  the  proper  match  for  a  given 
first-point  will  be  a  second-point  with  non-lrtefler  co-ordinates, 
Since  the  only  correlation  values  which  are  available  are  those  et 
Integer  second-oo I nts,  some  form  of  f nte r do  I  at  I  on  Is  necessary* 

Therefore,  the  final  operation  on  a  match  It  an 
Interpolation,  A  function  of  the  form  EXP(  -  (A*X*2  ♦  B*X  *  C»X*Y  + 
[*yt?  ♦  E*y  ♦  r ) )  |g  fitted  by  least  sauares  taohnlaues  to  the 
torrelatlon  values  between  the  window  around  the  first-point  and 
•imllar  windows  around  points  In  the  neighborhood  of  the 
«ecord-po i nt.  Solving  this  function  for  a  maximum  result*  In  either 
i  new  match  at  some  non-integer  second-point.  or  In  an  error  if  there 
*s  no  maximum  within  a  one-bixe)  radius  of  the  matching  seoond-po I nt. 

In  the  latter  case.  the  first-area  is  said  to  have  no 
reliable  ratch,  and  the  rrogr*m  continues  to  another  first-area,  Tha 
most  common  cause  of  sucn  failure  Is  a  strong  linear  edge  with  little 
Information  on  alther  sloe,  in  which  case  the  ohances  of  error  are 
sifflelent  to  cast  any  such  match  in  doubt, 

If  the  match  passes  this  final  test,  It  Is  recorded  for  use 
I-  a  later  program,  and  this  program  proceeds  to  another  first-area. 


CAMtFA 


The  job  of  this  program  Is  to  find  camera  models,  A  camera 
model  consists  of  seven  numbers  which  specify  the  focal  lengths  of 
the  twc  cameras  ana  the  orientation  of  the  second  oamer*  with  respect 
to  the  first,  \ 

The  first  camera  is  taken  to  have  its  fooal  point  at  the 
origin.  its  foeal  axis  along  the  z-axls*  and  Its  image  olene  the 
Dlar.»  z sF l ,  (See  Illustration  l.)  The  focal  point  of  the  second 
camera  Is  a  point  which  Is  described  by  the  baseline  distance  end  two 
angles,  The  two  angles  are  the  angles  by  which  the  first  camera  must 
be  Panned*  then  tilted*  to  point  at  the  second  focal  point,  <Sae 
Illustration  2.)  The  focal  axis  of  the  second  camera  Is  described  by 
two  more  angles.  They  are  the  angles  through  which  th*  first  camera 
must  be  panned,  then  tl!+ed  so  that  Its  axis  parallels  the  axle  of 
the  second  camera,  The  image  Plane  of  the  second  oema'a  Is  the  plane 


I  I  I ustrat ion  2, 


Co-ordinate  system  w  1 first  camera  canned  and  tilted  to 
focal  colnt  of  second  camera# 


perpend  I  cu  I  ar  to  t*>e  focnl  axis  at  distance  F2  from  the  focal  point, 
(See  Illustration  3.)  Tn9  orientation  of  the  second  image  Plane  is 
deserted  oy  the  angle  through  which  the  first  Image  plane  must  roll 
(aft®r  having  oeen  panned  and  tilted  to  make  the  axes  parallel)  In 
order  to  bring  tH9  two  "up"  directions  Into  agreement, 

This  program  takis  as  Innut  a  set  of  pairs  of  flrst*polnts 
and  s^cond-polnt-s  found  to  be  matches  and  attempts  to  find  a  camera 
rods l  which  would  account  for  these  Doint-nairs.  Determination  of  a 
model  Is  done  by  minimizing  a  measute  ..of  camera  model  error. 

The  error  measure  Is  the  average  error  In  match  taken  over 
the  pol'-'t  pairs,  Tor  each  point  pair#  the  error  In  match  Is 
aet9 'm  I  r  <?d  as  follows:  Using  the  camera  model#  the  first-point  Is 
projected  Into  space#  yielding  a  ray  from  the  focal  point  through  the 
Image  point,  This  ray  's  then  oaok-projected  Into  the  Image  Plane  of 


Illustration  3 , 

Co-o r d I rat°  system  with  second  camera  In  place#  first  camera  panned 
and  tilted  so  Its  fecal  axis  parallels  that  of  the  second  camera. 
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s | d i n3  a  | i n© 
t°  tie  the  saua.o 
second-op ! nt  and  this  line  segment, 
(See  Illustration  4 , ) 


s  taken 


segment  in  the  second  l^age,  The 
e  of  the  distance  between  th® 
in  the  usual  mathematical  sense. 


Actual  minimization  of  the  error  function 
the  1 nmenendent I y  comoiled  subroutine  M I N I M E • 


s  carried  out  by 
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Illustration  4,  # 

Co-ordinate  system  with  both  cameras  in  place,  showing  a  first-point 
projected  Into  soace  and  the  resulting  ray  back-projected  Into  the 
second  I r  a i e . 


This  suo-Program  Is  a  function  nlnlmlzer  which  uses  no 
<  e  r  i  ve  t  J  vq  information  In  seek  I  no  a  minimum,  Not  using  derivative 
information  was  a  constraint  forced  by  three  cons  I  derat | ons--the  fact 
that  thf>  derivatives  of  the  camera  model  error  funotlon  are 
i  i  scpnt  i  nuous  sjnce  the  function  has  been  truncated  to  avoid 
*  loading  point  ov9rficwS  in  tne  calculations,  the  fact  that  the 
locations  of  these  discontinuities  are  not  precisely  known*  and  the 
fact  that  th**  camera  model  error  function  itself  does  not  obey  any  of 
tie  constraints  ( monoton i c i ty,  lack  of  local  minima*  ete.)  usually 
Dlaced  on  functions  to  b®  minimized  by  derivative  methcds, 

The  workhorse  function  of  this  sub-program  takes  an 
n«d I  mens  I ona I  vector  (fn  the  camera  model  case,  n  =  7)  and  finds  3 
starting-points  elons  this  vector  such  that  an  upward-facing  parabola 
can  he  fitted  to  the  thr°e  points.  The  Inner  loop  fits  the  parabola, 
fiics  tne  minimum  of  this  narabola*  evaluates  the  function  at  the 
naranole  minimum,  and  chooses  wnich  of  the  four  available  DOlnts  are 
"best"  to  fit  anotner  parabola  to.  This  continues  elthar  until  the 
specified  number  0*  cycles  (usually  12)  have  been  completed  or  until 
succ°ss|ve  parabola  fits  yield  the  sane  function  value,  within  a 
tolerance  (usually  .PEhJJl), 

Tne  outer  loop  of  this  sub-orogram  constructs  a  set  of 
orthonorr-a  l  vectors  (starting  w|th  the  co-ordinate  axes)  and  calls 
tne  workhorse  funct|on  described  above  along  each  of  these  vectors, 
uher  this  set  of  vect0rs  Is  exhausted,  the  outer  loop  then  calculates 
tne  vector  difference  between  the  starting  point  for  that  round  and 
the  final  oolnt  f 0 u r  d ,  This  vector  Is  then  used  In  constructing  a 
new  set  of  orthonormal  vectors,  This  iteration  continues  until  the 
difference  In  starting  and  finishing  function  values  Is  less  than  the 
dive-  tolerance  <as  above )  or  until  the  function  value  drops  below 
some  pre-set  I Iml t. 


Like  all  m I r i m I ze r s *  this  one  can  get  stuck  at  a  local 
minimum,  Tnls  Is  jnfortunate.  fcecause  the  camera  model  error  funotlon 
act ea r s  to  have  »  Urge  nunoer  of  Potential  local  minima  near  the 
actual  minimum,  H0wever,  it  also  apDears  that  having  more  points 
available  for  use  reduces  the  number  and  depth  of  local  minima, 
reducing  tie  chance  of  a  spurious  minimum  being  taken  as  the  actual 
minimum,  Tnls  suggests  iterating  Cameka  w  i  th  the  following  Drogram, 
Camsch,  to  3et  better  arc  better  camera  models  based  on  more  and  more 
o  0  I  r.  t  s , 


Cav$Ch 


Reproduced  from 
besl  available  copy. 


This  program  takes  as  Incut  e  camera  model  either  derived  by 
CA/L-A  or  supplied  py  some  other  means,  To  match  a  designated  flrst- 
00 1  r.t,  this  program  projects  the  fir  st-oo  I  nt  Into  space. 


back-proJects  the  ray  s©  produced  lnt©  the  seoond  lmage#  than 
searches  along  the  result*^  Mna,  (See  Illustration  5,) 

The  actual  search  consists  of  stepping  along  the  line# 
starting  from  the  Infinity  point  (the  point  of  the  line  corresponding 
to  the  point  on  the  projected  ray  which  Is  farthest  away  from  the 
first  camera#  but  still  visible  to  the  second  camera)#  At  e'aoh  step 
along  the  line#  the  correlation  between  the  window  around  the 
flrst-p0!-t  under  eons I d e r a 1 1 0 n  a nd  the  c0r resD0nd I nfl  wl  d0w  aPound 
the  .econd-oolnt  currently  und«r  scrut,ny  's  calcul»ted*  $t®DD*nO 
continues#  the  best  N  such  correlations  (again#  N  Is  usually  5)  are 
kept  track  of  and  a  local  search  for  maximum  correlation  Is  done  In 
the  neighborhood  of  each  such  second-po 1 nt , 

Testing  whether  or  not  the  match  Is  sufficiently  good  and 
Interpolation  are  similar  to  the  same  processes  described  for  PARSET, 

One  search-pruning  heuristic  used  In  both  PARSET  and  CAMsCH 
neecs#  perhaps#  to  be  justified#  In  PARSET#  If  the  centar  point  of  a 
given  second-area  does  not  show  a  positive  correlation  with  the 


I  I | ustrat Ion  5 , 

A  pair  of  stereo  views 
5howlng  a  point  and  Its 
surrounding  correlation 
wlncow  overlaid  on  the 
too  I  mage  and  the  | I ne 
that  the  point  projects 
to  overlaid  on  the  lower 
Image,  Polaroid  picture 
taken  of  Data  Disc  while 
CAMSCH  was  In  operation 
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center  rolnt  of  tie  first-area,  the  second-area  is  rejected  as  being 
an  improbable  Place  tc  '  ook  for  a  -natch,  CAMSCH,  Instead  of 
exapinlr,-  every  r  c  i  n  t  a|t>m  tne  line  projected  Into  the  second  imaoe, 
<?x?.rr  infis  every  N-th  point,  where  N  is  half  the  radius  of  the 
correlation  winoow  cel  rg  useo.  Thus*  in  both  programs,  decisions  are 
•^ade  O"  the  basis  of  the  value  of  t -» e  correlation  function  at  some 
second-point  near  (but  net  at)  the  matching  second-point,  This,  it 
turns  out,  is  a  reasonable  thing  to  do,  Taking  any  cross-section  of 
thP  correlation  function  will  yield  a  graoh  shaped  somewhat  like 
£ X p ( - X t c ) ,  because  c f  this,  a  correlation  at  a  second-point  near  the 
match  will  ne  feir|y  njph--at  least,  aoove  the  noise  level,  Hence  a 
ver^  low  correlation  value  can  be  taken  to  mean  that  there  Is  no 
matcr'  in  the  vicinity  of  the  point  under  consideration,  and  the 
computation  necessary  to  search  that  area  can  be  avoided. 

It  is  interesting  to  note  that  different  programs  require 
n  i  f  f  e  r  p  r  t  d  e  *t  r  *  e  s  of  accuracy  from  their  camera  models, 


CAMSCH, 

met  h y  with  rathe 
will  rut  CAMSCH 
m  3  t  c  r  e  S  *or  most 
camera  ,t  o  a  9  I  ha  v  1 
enough  for  -n  tch  I 
suncest  not  spo 
camera  model,  as 
an  inaccurate  r 
fine  more  Points, 
etc. 


for  Instance*  with  its  local  search  strategies*  can 
r  jnpccurate  camera  models,  Any  camera  model  which 
in  the  rignt  ballpark  is  good  enough  to  oroduce 
points,  f'xoer  imentat  i  on  nas  shown  that  almost  any 
nr  an  average  squared-error  below  ,25  pixels  Is  flood 
n&  ?t  least  53%  of  the  points  tried,  This  would 
nciinG  great  amounts  of  time  minimizing  the  first 
nas  open  cone  |n  most  cases  tried  so  far.  Instead, 
coel  can  be  derived  quickly,  CAMSCH  can  use  this  to 
Which  C&n  be  used  to  derive  a  better  camera  model, 


Cn  the  other  hand,  programs  whi-h  do  depth  modelling  require 

6  #  t  r  *me  |  y  accurate  camera  riddels,  Tor  instance,  on  one  pair  of 
nictures,  about  £5  different  camera  models  were  derived,  Their 
squared -  error  varie-j  frCm  .2  to  ,0-404,  Oeoths  given  for  one  point  at 
- 9 ; s u r s c  distance  ZIC  feet  from  tha  camera  ranged  from  25  feet  to 

7  -  '■  In  9  n  e  r  f,  I ,  -odels  with  smaller  squared-errors  were 
setter  than  those  with  larger  errors,  However,  the  best  model  was 
net  tha  one  with  the  lowest  error!  Tnis  would  indicate  that  the 
r-odeis  telng  found  all  represent  local  minima  on  the  error  function; 
the  true  model  h ? s  ,v°t  to  re  found, 


This  result  leu  *0 
r.o:e'  c  f  h  e  I  n  o  *0  1  9  to  <■?  e  t 
is  ret  possible,  AcCJrj1t9 
0  =  r’ e  ” d  I  r  <  on  w 0 e  t ;n e  r  cr 
enough, 


further  moclfication  of  the  ninlmlzer*  In 
closer  to  the  reel  minimum,  As  yet,  this 
depth  ranging  remains  a  hlt-or-miss  thing, 
not  a  model  can  be  found  which  (s  good 


Ther*  sr9  several  other  minor  Programs  Intended  for 
0  h~  ~  r  s  t  r  *■  1 1  0  ns  which  fit  into  r.hls  set  of  Programs, 

ir-gno  s  i  -o  I  y  trkes  -•  f  '  I  e  containing  polot-Dairs  and 
displays  ti’yi,  0?.  1  r  ?.t  a  tine,  as  overlays  on  pictures  shown  on  Data 
Lise. 


DEPTH  works  similarly  to  WINSHO,  except  that  It  also  asks  for 
a  camera  model*  and  calculates  the  distance  from  the  first  camera  to 
the  three-dimensional  eolnt,  It  then  displays  this  distance  on  the 
Data  Disc  screen  with  the  appropriate  overlays  represented  the  two 
points,  At  the  end?  this  program  rounds  each  depth  to  the  nearest 
unit  of  distance  and  displays  these  distances  as  overlays  at  the 
corresponding  points  in  the  first  picture,  (See  Illustration  6.) 


I  1  | ustrat 1  on  6 , 

Photograph  from  Data  Disc 
of  overlaid  depths  as 
generated  by  DEPTH, 


CONCLUSION 

There  are  a  number  of  possible  variations  on  this  set  of 
programs , 

One  possible  alteration  would  be  to  Incorporate  oolor 
Information  Into  th9  metchlng  process,  This  change  would  reoulre 
using  three  aligned  color-filter  pictures,  and  modifying  the 
correlation  function  so  that  It  uses  a  vector  of  Information  at  each 
cloture  point  rather  than  a  single  scalar  number.  This  teehnlaue 
would  give  more  Information  at  each  oolnt,  making  gross  mismatches 
less  likely, 
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Another  interesting  icea  would  be  to  use  the  camera  model  to 
sn-aDe  tr.P  correlation  window,  Pacing  correlation  less  sensitive  to 
nistorticr''  of  an  object  due  to  the  differences  In  projection,  Thus, 
a  oar-era  model  having  most  of  the  change  in  the  horizontal  direction 
woulr  tn  made  to  causa  j  correlation  window  which  was  tall  and 
t  n  i  r.  -  -  sj  s  i  n  c  more  inform  a  tion  in  the  direction  in  which  little 
distortion  occurs,  less  in  the  distorted  direction.  (See 
Illustration  7.) 

f  course,  much  can  pe  done  to  automate  these  programs.  At 
present  they  o  f  t  °  n  stoo  and  as^  for  guidance  when  there  Is  doubt  as 
to  whether  or  not  a  match  is  good. 

In  p  A  R  S  E  T »  where  all  macches  must  be  good,  strengthening  the 
c  r i t  °  r I c  o  for  a  match  Is  one  way  to  do  this--when  in  doubt,  throw  It 
out.  Another  method  of  insuring  tnat  CAMERA  gets  only  flood 
noir. t-pairs  Is  to  allow  CA^EFA  to  weed  point-pairs  given  it.  If, 
when  a  minirum  of  sorts  is  found,  one  or  more  of  the  polnt-oalrs  are 
four.n  to  contribute  abnomrlly  high  (or  low)  errors,  these  pairs 
coulc  be  r&j^cteo,  and  a  r c -m i n  1  m i zat I  on  done, 

Theoret i ca l I y--that  is,  given  enough  time  and  some  low 
c  !  5v»  rne'-s- 1 1  is  possible  to  us«  CAMSCH  to  find  a  matching  point  for 
*vrry  f  i '■st-no  I  nt  which  has  a  match.  Under  this  assumption,  the 
terhniou®  of  par?|  taxing  (creating  mappings  from  one  picture  to 
•j  n  o  t  r  e  r  >  and  fir.dinS  oa.ra|,ax  edges  is  now  feasib|3,  Assumlnfl  flood 
camera  rodelg,  ths  possess‘on  of  such  a  mapping  makes  dept1"1  modelling 
and  tne  location  of  ceoth  edges  p  o  s  s  i  D  1  e  . 

It  seems,  tw  =  n  that  the  next  moves  on  this  project  will  be  In 
tr,  direction  of  improvements  to  existing  programs,  Specifically, 
tot  low  running  necessary  to  form  the  matches  needs  to  be  developed. 
t w p  programs  d  o  c  u r1  e  n  t  e  a  here  need  t  c  be  optimized  so  they  can  run  In 
^n  a^oup-  of  timn  agreeable-  to  the  machine.  Most  of  all,  the  camera 
'-'Ode  i  programs  need  to  oe  improved  so  that  more  reliable  camera 
more I s  are  possible. 


Illustration  7, 

Various  "seccnc-c  !  ctures"  snowing  hacK-p ro Jected  first-points  and  the 
a  p  r  r  o  p  r  i  a  t  e  l  v  she.  pec  correlation  windows  (size  exaggerated). 
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INTRODUCTION 


My  project  for  this  Quarter  was  to  start  ! mp | ementat I  on  of  a 
system  for  processing  color  etereo  pairs,  simitar  to  my  system  for 
process  I  no  blaok  and  white  stereo  Images  (see  Hannah,  1971,  for 
detalle  of  that  system),  Slnoe  the  blaok  and  white  system  was  built 
around  the  Idea  of  using  normalized  cross-oor re lat I  on  as  a  measure  of 
match  between  two  points,  the  first  thing  that  was  needed  for  the  new 
system  was  some  egulvalent  measure  of  matoh  for  color  Images. 
(Actually,  correlation  Is  a  measure  of  matoh  between  two  areae)  the 
vwo  points  refered  to  are  the  centers  of  the  areas,  In  the 
following,  the  phrases  "between  two  points"  and  "between  two  areas" 
will  be  ueed  Interchangeably  In  referring  to  correlation), 

Basically,  I  had  the  choice  of  somehow  altering  correlation 
for  use  with  color  Information  or  creating  an  entirely  different 
oeasure  of  match,  Having  had  little  luck  In  an  earlier  attempt  to 
find  a  new  measure  of  match  for  the  black  and  white  case,  I  chose  to 
oodlfy  correlation. 

This  document  reports  the  derivation  and  Implementation  of 
color  correlation,  and  describes  a  program,  NEWPTS,  which  finds 
Initial  point-pair  matches  In  either  color  or  black  ar.d  white  stereo 
pairs, 


COLOR  CORRELATION 

It  Is  generally  recognized  that  oolor  conslete  of  three 
components,  A  child  learns  in  grade-eohool  art  that  all  oolore  can 
be  made  from  red,  yellow,  and  b | ue  pigments,  In  hlgh-eohool  physios, 
he  Is  told  that  all  colors  result  from  red,  green,  and  b I light. 
In  college  psychology  courses,  color  Is  discussed  In  terms  of 
Intensity,  hue,  and  saturation, 

Ignoring  for  the  moment  the  thorny  Questions  of  what  the 
components  of  oolor  "really"  are,  we  shall  admit  only  that. there  are 
three  such  components,  Since  the  color  Images  we  currently  are 
working  with  were  obtained  by  digitizing  three  black  and  white 
pictures  which  resulted  from  photographing  an  ordinary  color  elide 
under  red,  green,  and  blue  filters,  respectively,  we  shall  refer  to 
the  components  as  R,  G,  and  B, 

It  is  somewhat  more  convenient  (as  well  as  more 
mathematical:)  to  think  of  a  oolor  plotur*  ae  one  array  of 
vector-valued  points  (r,g,b)  Instead  of  three  separate  arrays  of 
scalar-valued  points  r<  g,  and  b,  This  suggests  regarding  the 
text-book  verelon  of  normalized  cross-oorre  latlon 


SUM (  (X-MEAN(X))  *  (y-MEAN(y) )  ) 

COR  . . . . . . - . - . 

SQRT (  SUM (  (x-MEAN(x) )»2  )  *  SUM(  (y-MEAN(y) ) *2  )  ) 


(wher®  small  letters  denote  sample  elements*  SUM  Is  the  sum  over  some 
set  of  such  elements»  MEAN  Is  such  a  sum  divided  by  the  number  of 
elements  summed  oven  SQRT  Is  the  square  root  function*  and  *  denotes 
mu  1 1 1 b I  I  cat  I  on) 


as  the  one-d Imens I ona |  ease  of  a  vector  function 


VCOR 


SUM {  (X-MEAN(X))  •  (Y-MEAN(Y))  > 

SORT (  SUM<  IX-MEAN(X) | *2  >  *  SUM(  I Y«MEAN( Y> I t2  )  ) 


(where  capital  letters  denote  veotors,  •  Is  vector  dot  product*  and 
I A I  Is  the  norm  of  the  vector  A), 


Considering  only  the  factor  SUM<  (X-MEAN(X))  •  (Y-MEAN(Y))  ) 
<slnce  SUM (  |X-MEAN(X> |t2  )  and  SUM(  | Y«MEAN( Y) | *2  )  are  both  special 
cases  of  this  sum  with  x  substituted  for  Y  In  the  first  case  and  Y 
substituted  for  X  In  the  seoond)  and  letting  x  be  (xr,xg,xb>  and  Y  be 
(yr*yg*yb)!  we  have 


SUM (  ( X-MEAN  <  X )  )  e  <  Y-MEAN ( Y  ) )  ) 


s  SUM <  ( < «r * xg, xp)-MEAN<  (xr*xg#xb)  >)  e 

< <yr»yfl*yb)-MEAN(  (yr»yg»yb)  ))  ) 

»  SUM <  <xr-MEAN<xr)»xg«MEAN(xg)»xb-MEAN(xb) >  * 

<yr-MEAN(yr)*yg-MEAN(yg>,yb-MEAN(yb> )  ) 

5  SUM (  ( xr-MEAN< xr ) )*(yr-MEAN(yr > )  *  (xg-MEAN(xg) >*<yg«MEAN(yg) )  * 

(xb-MEAN< xb) )*<yb-MEAN(yb) >  ) 

If  we  clever|y  notloe  that  all  three  terms  within  this  eum 
are  the  same  In  form  and  combine  them  Into  one  term  under  a  summation 
which  sums  over  all  components  as  well  as  all  elements  of  oomponents, 
we  get 

*  SUM (  (x-MEAN(x>)  *  (y-MEAN(y) )  > 

which  Is  the  representative  faotor  of  the  formula  for  ordinary 
correlation  (see  above  formula  for  correlation), 

It  Is  convenient  (If  somewhat  embarasslng)  to  have  oolor 
correlation  turn  out  to  be  a  dressed  uo  form  of  ordinary  correlation, 
for  this  means  that  color  correlation  has  all  of  the  mathematical 
properties  of  ordinary  correlation,  This,  In  turn,  will  be 
particularly  useful  In  the  Interpolation  of  correlation  values  at 
non-integer  points  In  the  picture,  since  previously  developed 
techniques  for  such  Interpolation  need  not  be  Justified  again, 

Since  both  vector  and  scalar  correlation  have  the  same  form, 
save  for  the  number  of  components  whjoh  need  be  summed  over*  the  two 
brands  of  correlation  have  been  Implemented  as  one  subroutine, 
CORLAT,  In  the  Sail  load. module  SCOREL,  Which  calculation  Is  done 


depends  on  the  global  flag  COLOR,  For  expedlenoy  In  computat I  on,  the 
ooefflolent  Is  oaloulated  as 

<  n  *  SUM<  x  *  y  )  -  SUM ( x )  *  SUM(y>  )t2 

<  n  •  SUM<  x*2  )  -  SUM<x>*2  )  •  (  n  *  SUM<  y*2  )  •  SUM<y>f2  > 

that  Is*  with  sums  arranged  so  that  only  one  pass  need  be  taken  to 
calculate  all  sums,  Note#  too*  that  the  sauare  of  the  correlation  Is 
used  (as  It  was  In  the  blaok  and  white  case),  trading  a 
multiplication  for  a  oa|l  on  SORT, 

No  mod  I f I  cat  I (other  than  a  small  amount  of  optimizing) 
have  been  made  on  the  funotlons  MATCH  and  MAXCOR  whloh  o a  I  I  CORlAT 
and  live  In  SCOREL,  A  separate  program,  NEWPTS*  has  been  created  to 
serve  the  function  of  PARSET  In  the  color  case  and  replace  PARSET  In 
the  black  and  white  case,  It  Is  desorlbed  below  as  If  It  operated  In 
eolor  modei  only, 

NEWPTS 


The  purpos#  of  this  program  Is  to  find  a  set  of  pairs  of 
polntsi  one  point  out  of  eaoh  picture,  which  matoh,  Intuitively,  two 
points  matoh  If  they  both  are  orojeotlone  of  the  eame  three- 
dimensional  point,  Since  the  computational  orooess  oan  only  say  that 
point  A  matohes  point  B  with  some  probability*  this  program's  purpose 
Is  to  find  Pairs  of  points  which  matoh  with  fairly  high  probability, 

As  a  Preliminary  to  the  matching  procese,  this  program 
segments  both  pictures  Into  overlapping  areas,  usually  20  pixels 
eouare,  It  then  computes  the  mean  and  varlanoe  of  eanh  area  In  the 
first  component  of  each  oolor  picture  (only  one  oomoor^nt  1e  used  to 
expedite  computation)  and  sorts  each  oloture'e  areas  by  varianoe, 
keeping  track  of  where  In  the  picture  each  area  oame  from, 

The  matching  process  begins  by  seleotlng  an  area  at  random 
from  the  top  end  of  the  variance  list  of  the  first  oloture,  usually 
the  top  25%,  This  limitation  Is  Imposed  because  the  measure  of  matoh 
being  used  works  best  where  there  Is  a  la^fle  an,our't  of  Information 
present*  which  Is  symptomlzed  by  the  varlanoe  being  large, 

Since  areas  whloh  matoh  should  have  similar  variances*  the 
selected  area  of  the  first  picture  Is  oompared  with  each  area  of  the 
second  picture  whose  variance  Is  within  20%  of  that  of  the  area  under 
consideration,  (In  the  following,  let  the  prefixes  ’’first-”  and 
"seoond-"  stand  for  the  modifying  phrases  ”of  the  first  picture"  and 
"of  the  eeoond  picture"  respectively,) 

Each  eligible  second-area  Is  Initially  tested  to  see  If  Its 
mean  Is  similar  that  of  the  first-area,  If  a  second-area  passes  this 
test*  a  search  Is  made  to  find  the  second-point  (some  point  In  or 
near  the  second-area  under  consideration)  such  that  the  2n+l  x  2n+l 


window  surrounding  this  second-point  Is  the  best  match  for  (has  the 
highest  normalized  cross-corro I  at  Ion  with)  the  2n*l  x  2n+l  window 
surrounding  the  center  point  of  the  first-area-  The  searoh  strategy 
used  Is  essentially  that  used  by  Quam  ( 19  7 1  >  • 

For  computational  expediency!  the  above  searoh  Is  carried  out 
using  only  the  first  component  of  the  color  picture,  As  the 
orogram  proceeds  through  the  seoond-areasi  the  N  seoond-areas  (where 
N  Is  usually  5)  yielding  the  highest  correlation  values  are  kept 
track  of,  Later,  a  second  search  Is  done  on  these  areas  using  color 
correlation  to  determine  which  of  the  areas  that  matohed  on  the  basis 
of  the  one-oomponent  search  match  best  In  oolor. 

Tests  are  then  made  to  determine  whether  the  best  match  found 
was  good  enough,  First  of  all,  the  oorriiatlon  must  be  above  ,5 
(calculated  sauare  of  the  correlation  above  ,25)  t  since  a  correlation 
4*o we r  than  ,5  can  occur  between  areas  which  do  not  really  matoh. 
Secondly,  the  too  two  correlations  must  differ  significantly. 
Failure  to  do  so  would  Indicate  that  more  than  one  match  was 
possible,  oastlng  doubt  on  the  validity  of  either  match,  Failure  In 
either  of  these  tests  oauses  a  first-area  to  be  rejeoted  as  having  no 
reliable  match,  and  another  f|rst-ar«a  Is  tried, 

Note  that,  when  this  process  Is  finished,  the  center  point  of 
the  first-area  has  been  paired  with  a  second-point  whloh  has  Integer 
co-ordinates,  in  practice,  however#  the  proper  match  for  a  given 
first-point  w|||  be  a  seoond-polnt  with  non-integer  oo-ordlnates , 
Since  the  only  correlation  values  which  are  available  are  those  at 
Integer  second-points,  some  form  of  Interpolation  Is  neoessary, 

Therefore,  the  final  operation  on  a  match  Is  an 
interpolation,  a  funotlon  of  the  form  EXP(  -  (A*X»2  +  B*X  ♦  C*X*Y  ♦ 
q#Y*2  ♦  E*Y  ♦  F))  Is  fitted  by  least  squares  techniques  to  the 
color  correlation  values  between  the  window  around  the  flrst-oolnt 
and  similar  windows  around  points  In  the  neighborhood  of  the 
second-point,  Solving  this  funotlon  for  a  maximum  results  In  either 
a  new  match  at  some  non-integer  second-point,  or  In  an  error  If  there 
Is  no  maximum  within  a  one-p|xe|  radius  of  the  matohlng  second-point. 

In  the  latter  case,  the  first-area  Is  said  to  have  no 
reliable  match,  and  tha  Program  continues  to  another  first-area,  The 
oost  common  oause  of  such  failure  Is  a  strong  linear  edge  with  little 
information  on  either  side,  In  which  case  the  ohances  of  error  are 
sufficient  to  cast  any  such  match  In  doubt, 

If  the  match  passes  this  final  test,  It  Is  recorded  for  use 
in  a  later  program,  and  this  program  proceeds  to  another  flrst-are*. 


ADDITIONS  AND  IMPROVEMENTS 

Our  present  practice  of  using  red,  green,  and  blue  as  the 
three  components  of  the  color  picture  has  Its  drawbacks,  One  would 


J 


4<k8  to  oalculate  the  means  and  varlanoes  of  ths  average  Intensities 
in  the  thr®«  components  for  the  ourpoa®  of  narrowing  the  aaarchaa  far 
th«  Initial  oolnt-oalr  matchings,  To  do  th I s  undar  the  oreaent 
scherre»  one  muet  either  oaloulate  the  avaraoaa  as  one  ooaa--a  rather 
slo*  orooess--or  Ks#p  around  an  extra  pair  of  clotures,  the  Intensity 
pictures--®  scheme  which  enlaroens  (hence  alow®)  one's  Job 
excessively,  a  solution  to  this  problem  would  be  to  have  the 
ntsnslty  plcturs  be  one  of  the  three  components  of  the  oolor 
cloture, 

There  are  at  least  two  sohemes  of  oolor  representation  whloh  J 

have  Intensity  as  one  component,  The  best  known,  perhaps.  Is  ths 
Intensity,  hue,  and  saturation  soheme,  Commercial  television  uses 
the  different,  although  related,  scheme  of  Intensity,  x-,  and 
y-chromatlcity,  Both  of  these  ars  baaed  on  the  Idea  of  a  oo|op 
wheel,  I  ® ,  oolors  arranged  circularly  around  a  hub,  The 

hue-saturation  scheme  corresponds  to  using  polar  co-c-d 1  nates  to  ^ 

nocat®  a  oolor  Point  on  ths  wheel,  The  x»,  y-ohromat I c 1 ty  soheme 
corresponds  to  using  a  (not  necessarily  reotanoular)  Cartesian 
oo-ordlnate  system  to  locate  the  color  point, 

Implementing  one  of  these  systems  seems  to  be  desirable. 

Precisely  which  one  to  Implement  and  how  to  derive  these  components 
from  the  given  red*  or##n  and  blue  oomoonents  Is  a  matter  for  furthsr 
study, 

Onoe  the  r«o r esentat I  on  auastlon  has  besn  sattlsd,  there  art 
a  number  of  statistics  which  can  be  ~«!ou|ated  over  the  segmented 
oleturas.  In  addition  to  the  mean  and  wlcnoe  of  th#  Intensity,  one 
coulo  calculate  the  mean  and  variance  of  the  oolor,  the  mode  (most 
frequently  occurlnfl)  color,  possibly  the  next  most  frequent  color, 
etc,  Etch  area  In  each,  picture  would  then  be  associated  with  e  veotor 
of  suoh  statistics,  -*nd  searohss  for  a  matohlng  seoond-area  oould  be 
constrained  to  thosie  eecond-ereas  whose  vector  dletanoe  from  the 
flrst*#r#a  Is  within  soms  tolerance,  & 

A  more  thorough  I nvaet I  Oat  I  on  of  the  properties  and 
reprssantat Ions  of  color  saams  to  p®  In  order.  It  is  In  this 
direction  that  this  project  will  proceed. 


5b 


■1 


BIBLIOGRAPHY 

Hannah,  M,  J,  Ci97i],  "Camara  Models  and  Sterao  Image  Processing", 
Term  Project  Report#  CS-293#  Stanford  University*  fall.  1971, 

Quarr,  Lynn  H,  C 1 97 1 3 #  "Computa'  Comparison  of  Ploturee",  Stanford 
Artificial  intelllflence  Project  Memo  No,  144, 

% 


6b 


